All Alerts from Prometheus are opened with Urgency high

#1

Hi all,

I have a problem, because all alerts coming out of Prometheus are opened with urgency HIGH, even if the Alert form Prometheus has the Severity setting set to warning or info. Is somebody here who can give me a hint regarding this?

Thanks,
Volker

#2

Here a detailed view on the alert:
I wonder why Severity say ERROR? While it says in text below that severity is set to WARNING.

12
Screenshot 2022-05-03 at 19.17.12.png874×894 77.5 KB

#3

The severity field inside of the custom_details object is not the correct field to use with our severity-based urgency feature. You’ll need to set the proper severity field in the event API v2 payload (https://developer.pagerduty.com/api-reference/b3A6Mjc0ODI2Nw-send-an-event-to-pager-duty), or use the value in custom_details and create event orchestration rules to set the proper severity field for you and then use severity based urgency.

1 Like
#4

This is an example alert manager config that should work to set the proper severity field (hardcoded to warning in this case):

alertmanager:
  enabled: true
  config:
    global: 
      pager_service: "service_key"
    route:
      group_by: ['...']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'pagerduty_service_new'
      routes:
      - receiver: 'pagerduty_service_new'
        match_re:
          receiver: 'staging'
    receivers:
    - name: 'null'
    - name: 'pagerduty_service_new'
      webhook_configs:
        - url: 'https://api.pagerduty.com/'
      pagerduty_configs:
      - routing_key: {{ global.pager_service }}
        class: "{{ range .Alerts }}{{ .Labels.alertname }}{{ end }}"
        group: "{{ range .Alerts }}{{ .Labels.namespace }}{{ end }}"
        component: "{{ range .Alerts }}{{ .Labels.pod }}{{ end }}"
	    severity: 'warning'
        details:
          container: "{{ range .Alerts }}{{ .Labels.container }}{{ end }}"
          region: "{{ range .Alerts }}{{ .Labels.region }}{{ end }}"
          severity: "{{ range .Alerts }}{{ .Labels.severity }}{{ end }}"
          ownership: "{{ range .Alerts }}{{ .Annotations.ownership }}{{ end }}"
          summary: "{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}"
          runbook_url: "{{ range .Alerts }}{{ .Annotations.runbook_url }}{{ end }}"
          description: "{{ range .Alerts }}{{ .Annotations.description }}{{ end }}"
1 Like
#5

Thank you! I will try it.